Artwork generated to convey digital messages, and methods/apparatuses for generating such artwork

ABSTRACT

2D machine readable symbologies are stylized and made aesthetically-appealing, facilitating their use to convey plural-symbol data on product packaging and other articles. In some arrangements, symbologies are stylized by geometric transformations (e.g., by multiple rotation and/or mirroring operations) to develop tiles having organized geometric structures. Such stylized symbologies can be decoded by existing code readers. A great variety of other features and arrangements are also detailed.

RELATED APPLICATION DATA

This application claims priority to provisional applications 62/972,522,filed Feb. 10, 2020; 62/966,510, filed Jan. 27, 2020, 62/946,732, filedDec. 11, 2019, 62/916,021, filed Oct. 16, 2019, 62/841,084, filed Apr.30, 2019, and 62/824,934, filed Mar. 27, 2019.

This application is related to, and builds on the disclosures of,application Ser. No. 16/212,125, filed Dec. 6, 2018 (published as20190213705), Ser. No. 16/435,164, filed Jun. 7, 2019 (published as20190378235), Ser. No. 15/072,884, filed Mar. 17, 2016 (published as20170024840), and Ser. No. 16/405,621, filed May 7, 2019 (published as20190332840).

The disclosures of the above applications are incorporated herein byreference.

TECHNICAL FIELD

The present technology concerns message signaling. The technology isillustrated in the context of message signaling through artwork ofgrocery item packaging, to convey plural-bit messages—of the sortpresently conveyed by UPC barcodes. However, the technology is notso-limited.

BACKGROUND AND INTRODUCTION

Barcodes are in widespread use on retail items, but take up package realestate that the manufacturers would prefer to use for other purposes.Some retail brands find that printing a barcode on a product detractsfrom its aesthetics.

Steganographic digital watermarking is gaining adoption as analternative to visible barcodes. Watermarking involves making subtlechanges to packaging artwork to convey a multi-bit product identifier orother message. These changes are commonly imperceptible to humans, butare detectable by a computer.

FIGS. 1A and 1B shows illustrative digital watermark patterns, includingmagnified views depicting their somewhat mottled appearance. In actualuse, the watermark pattern is scaled-down in amplitude so that, whenoverlaid with packaging artwork in a tiled fashion, it is an essentiallytransparent layer that is visually imperceptible amid the artworkdetails—just slightly altering local image luminance or chrominance.

Designing watermarked packaging involves establishing a tradeoff betweenthis amplitude factor, and detectability. To assure reliabledetection—even under the adverse imaging conditions that are sometimesencountered by supermarket scanners—the watermark should have as strongan amplitude as possible. However, the greater the amplitude, the moreapparent the pattern of the watermark becomes on the package. A bestbalance is struck when the watermark amplitude is raised to just belowthe point where the watermark pattern becomes visible to human viewersof the packaging.

FIG. 2 illustrates this graphically. When a watermark is added toartwork at low levels, human visibility of the watermark is nil. Asstrength of the watermark increases, there becomes a point at whichhumans can start perceiving the watermark pattern. Increases inwatermark amplitude beyond this point result in progressively greaterperception (e.g., from barely, to mild, to conspicuous, to blatant).Point “A” is the desired “sweet-spot” value, at which the amplitude ofthe watermark is maximum, while visibility of the watermark is stillbelow the point of human perception.

Setting the watermark amplitude to this sweet-spot value, when creatingpackaging artwork (e.g., using Adobe Illustrator software), is onething. Hitting this sweet-spot “on-press” is another.

All print technologies, being physical processes, have inherentuncertainty. Some print technologies have more uncertainty than others.Dry offset printing is one; it is notably inaccurate. (Dry offset isadvantageous in other respects; for example, it works well with thetapered shapes of plastic tubs, such as for yogurt and sour cream.)

Dry offset offers only gross control of dot size and other printstructures. For example, if a digital artwork file specifies that inkdots are to be laid down with a density of 15%, a dry offset press maydeposit a much greater density of ink, e.g., with 30% density.

Printer profiles exist to characterize such behavior. A profile for aparticular model of dry offset press may specify that artwork indicatinga 15% density will actually be rendered with a 25% density (e.g., a 10%dot gain). But there is a great deal of variation between presses of thesame model—depending on factors including age, maintenance, consumables,temperature, etc. So instead of depositing ink at a 25% density—asindicated by a printer's profile, a particular press may instead depositink at a 20% density. Or a 40% density. Or anything in between.

This uncertainty poses a big obstacle for use of digital watermarktechnology. Packaging artwork that has been carefully designed to setthe watermark amplitude to the point “A” sweet-spot of FIG. 2, mayinstead be printed with the watermark amplitude at point “B”, making thewatermark plainly visible.

Our patent publication 20110214044 teaches that, rather than attempt tohide a digital watermark signal in artwork, the payload data may beencoded as overt elements of the artwork. One example is a digitalwatermark in which the amplitude is set to a plainly human-visiblelevel. Another example is a 1D or 2D black and white barcode that isused as a fill pattern, e.g., laid down by a paintbrush in AdobePhotoshop software.

These techniques often do not prove satisfactory. As illustrated byFIGS. 1A and 1B, a digital watermark, with its amplitude set to aplainly human-visible level, yields a pattern that does not fit wellinto the design of most packaging. Painted barcode fills similarly havelimited practical utility.

In accordance with one aspect of the technology, a 2D machine-readablecode is stylized by a collage process, based on excerpts copied from oneor more artworks. The result no longer has the appearance of amachine-readable code, yet can be read by a compliant code readingapparatus. Such a code can be tiled and serve as a background forproduct packaging artwork, e.g., as shown by the coffee bean backgroundin the yogurt label of FIG. 3. Unlike the arrangements detailed inapplication Ser. No. 16/212,125 (published as 20190213705), embodimentsof the present technology typically do not employ a neural network.

In accordance with another aspect of the technology, a 2Dmachine-readable code is stylized by performing one or moretransformations on the code itself, imparting an aesthetic patterningthat can make the code suitable as a graphic design element, while stillencoding a plural-bit message. In some arrangements the transformationsalso serve to extend the circumstances in which the code is readable.

The foregoing and additional features and advantages of the presenttechnology will be more readily apparent from the following detaileddescription, which proceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIGS. 1A and 1B illustrate exemplary digital watermark patterns.

FIG. 2 is a graph showing how a digital watermark is human-imperceptibleat low strengths, but becomes progressively more conspicuous at higherstrengths.

FIG. 3 shows artwork for a yogurt container, including a backgrounddigital watermark pattern stylized with coffee bean artwork.

FIG. 4 illustrates a style image (depicting coffee beans) and awatermark image, which are processed according to the present technologyto generate a coffee bean college that conveys the watermark's payload.

FIG. 5 schematically illustrates the selection of blocks from a styleimage to mimic a watermark pattern by a tiled collage.

FIG. 6 depicts a matrix operation employed in an illustrative embodimentof the present technology.

FIG. 7 is a diagram showing adjoining edges that are considered as eachblock is added to a tiled collage, so as to minimize edgediscontinuities.

FIG. 8A shows a style image depicting concentric circles.

FIG. 8B shows a collage produced from the style image of FIG. 8A and awatermark image, considering block correlation, but without regard toedge artifacts.

FIG. 8C shows a collage produced from the style image of FIG. 8A and awatermark image, minimizing edge discontinuities, but without regard toblock correlation.

FIG. 8D shows a collage produced from the style image of FIG. 8A and awatermark image, which considers both block correlation and edgeartifacts.

FIGS. 9A and 9B detail Matlab code used in an illustrative embodiment ofthe present technology.

FIGS. 10, 11 and 12 show style images, and corresponding watermarkedcollages, in accordance with certain embodiments of the presenttechnology.

FIG. 11A shows a watermarked collage to which QR code synchronizationmarkings have been added.

FIG. 13 shows that a collage can be formed of two tiled, overlappingarrays of blocks selected from a style image.

FIG. 14A shows a weighting function.

FIG. 14B shows the weighting function of FIG. 14A applied to a region offour corner-adjoining blocks, and defining a complementary weightingfunction at a central region.

FIG. 14C shows two complementary weighting functions, which sum to unityat all points.

FIGS. 15A and 15B show two arrays of weighted blocks that can beoverlapped and summed to yield a collage.

FIG. 15C shows the collage resulting from FIGS. 15A and 15B.

FIG. 16A shows a style image, and FIG. 16B shows a correspondingwatermarked collage produced therefrom.

FIG. 17A shows another style image, and FIG. 17B shows a correspondingwatermarked collage produced therefrom.

FIGS. 18A, 18B and 18C detail Matlab code used in an illustrativeembodiment of the present technology.

FIG. 19A shows a weighting function.

FIG. 19B shows the weighting function of FIG. 19A applied to a region offour corner-adjoining blocks and defining a complementary weightingfunction at a central region.

FIG. 19C shows two complementary weighting functions, which sum to unityat all points.

FIGS. 20A and 20B show a border weighting function applied to twoblocks.

FIG. 20C depicts a combination of the two weighted blocks of FIGS. 20Aand 20B.

FIG. 21 illustrates a style image (depicting a denim fabric weave) and apreviously-watermarked image (Lena), which are processed according tothe present technology to generate a denim collage that depicts the Lenaimage including its watermark.

FIGS. 22A, 22B, 22C and 22D show the collage of FIG. 21 in which pixelvalues within each selected style block have been adjusted based on theaverage pixel value of the target image block to which that style blockcorresponds.

FIGS. 23A, 23B, 23C and 23D show another collage in which pixel valueswithin each selected style block have been adjusted to remove a fractionof the block's average value, and to add a fraction of the correspondingtarget image value.

FIGS. 24A-24I show different transformations that can be used inembodiments employing a further aspect of the present technology.

FIG. 25 shows a 2D code block that can serve as an input indicia towhich various transformations are applied.

FIG. 26 shows one such transformation of the FIG. 25 indicia, based on asummation of multiple rotations of sub-blocks of the input indicia.

FIG. 26A further illustrates the transformations employed in FIG. 26.

FIG. 27 shows another transformation of the FIG. 25 indicia, based on asummation of multiple mirrorings of sub-blocks of the input indicia.

FIG. 28 shows a further transformation of the FIG. 25 indicia, based ona summation of multiple shiftings (translations) of sub-blocks of theinput indicia.

FIG. 29 shows yet another transformation of the FIG. 25 indicia, basedon a summation of multiple mirrorings and FFT-shifts of sub-blocks ofthe input indicia.

FIG. 30 shows still another transformation of the FIG. 25 indicia, basedon rotations, negations, and FFT-shifts of sub-blocks of the inputindicia.

FIG. 31 shows a further transformation of the FIG. 25 indicia, based onrotations and mirrorings of sub-blocks of the input indicia.

FIG. 32 shows two processed signal indicia tiled side-by-side, where asequence of component tiles repeats.

FIG. 33 shows an excerpt from two processed signal indicia tiledside-by-side, where a sequence of components tiles does not repeat.

FIG. 34 shows a transformation of the FIG. 25 indicia based on hexagonalsub-blocks and a summation of multiple rotations of sub-blocks of theinput indicia.

FIG. 35 shows a transformation of the FIG. 25 indicia based on hexagonalsub-blocks and a summation of mirrorings rotations of sub-blocks of theinput indicia.

FIGS. 36A-36D show transformations of the FIG. 25 indicia based oncombinations of different transformations—all leading to the same outputpattern.

FIG. 37 shows a transformation of the FIG. 25 indicia based on asummation of 14 different rotations of the input indicia.

FIG. 38 shows a transformation of the FIG. 25 indicia based on asummation of 60 equally-spaced rotations of the input indicia, combinedwith a similar summation of 10 rotations.

FIG. 39 shows a transformation of the FIG. 25 indicia based on 60rotations—each at a different scale.

FIG. 40 shows a transformation of the FIG. 25 indicia based on 10,000rotations—each at a different scale.

FIG. 41 shows a transformation of the FIG. 25 indicia based on asummation of 21 differently-scaled versions of the input indicia.

FIG. 42 shows a transformation of the FIG. 25 indicia based on 20different rotations, each differently-scaled.

FIGS. 43A-43C illustrate different kaleidoscopic operations.

FIGS. 44A-44C illustrate different windowing operations used inconnection with the operations of FIGS. 43A-43C.

FIGS. 45A-45D further illustrate kaleidoscopic operations.

FIG. 46 shows a food container, such as a plastic yogurt tub, printedwith a code indicia produced by one of the present methods.

FIG. 47 illustrates how a stylized pattern produced by one of thepresent methods can be combined with host artwork.

FIGS. 48A-48C detail how the patterning produced by certain embodimentsof the present embodiment becomes less distinct with larger sub-blocksizes.

FIG. 49 is a flowchart detailing one particular implementationincorporating aspects of the present technology.

FIG. 50 illustrates that artwork according to aspects of the presenttechnology can include both a conspicuously-structured sub-block,together with seemingly-random elements.

FIGS. 51A-51E illustrate a few different sub-block arrangements that canbe employed in embodiments of the present technology.

FIG. 52 is a flow chart detailing another particular implementationincorporating aspects of the present technology.

FIG. 53 illustrates that an input indicia can be geometricallytransformed prior to being divided into sub-blocks, for application ofother transformations.

FIG. 54 details how an input signal block can be conformally-mapped intoa geometrically-transformed form, after which sub-blocks thereof aresymmetrically-transformed, and inverse-conformally mapped.

FIG. 55A shows that a patterned signal block may be made lighter bytone-mapping.

FIG. 55B shows that a patterned signal block may be made lighter byscreening.

FIG. 56 shows an example of a signal block formed from combining twodifferent resolution signal blocks.

FIG. 57 contrasts one aspect of signal rich art versus traditionalwatermarking.

FIG. 58 shows amplitude and frequency modulation of dot, Delaunay andVoronoi patterns.

FIGS. 59A and 59B show an excerpt from a greyscale illustration of areindeer, rendered in halftone form using Voronoi and Delaunay patterns,respectively.

DETAILED DESCRIPTION

Section I

An exemplary method according to one embodiment of the presenttechnology produces a stylized version of a digital watermark signal,based on an input digital watermark signal image (sometimes termed atarget image) and one or more style images (sometimes termed sourceimages). The stylized version of the watermark signal is composed as acollage (or mosaic or patchwork) from pixel patches excerpted from thestyle images. To a human observer, the stylized artwork is evocative ofthe style image(s), rather than the watermark signal (e.g., thebackground of FIG. 3 appears as a field of coffee beans). To a compliantdigital watermark decoder, however, the stylized artwork is interpretedas a signal carrier, conveying an encoded plural-symbol payload.(Digital watermark decoders mostly ignore the low frequency imagecomponents that dominate human vision, and instead tend to decodesignals from the high frequency pattern variations that elude humaneyes.)

The stylized image in a preferred embodiment (sometimes termed a mixedimage, or collage, or mosaic image) is created by correlating blocksfrom the watermark signal image with blocks from the style image (asingle style image is assumed for expository convenience). The styleimage block with the best correlation is substituted for the watermarksignal block, for each of the watermark signal blocks. A watermarkedpattern is thereby created without the conventional act of modulatingpixels in a host image.

Since the resulting payload-conveying stylized image is a patchwork ofthe unmarked style image, typical watermarking signal visibilitycriteria need not be considered. But a new visibility concernarises—edge discontinuity in the collage.

To reduce this concern, the watermark signal can be divided intorelatively fewer blocks of larger size, reducing the number of adjoiningedges in the resulting collage. As a practical matter, the selected sizeof the watermark signal blocks depends on attributes of the styleimage—particularly its spatial autocorrelation. If the style image has avery regular pattern, like parallel lines or a repetitive/tiledgeometric pattern, it will have a high spatial autocorrelation and maynot be suitable for creating a stylized watermark pattern. Other styleimages, with a bit less spatial autocorrelation, may be suitable,provided the blocks are small in size, e.g., with side dimensions of1-3% of the side dimensions of the watermark signal image. Better arestyle images with less autocorrelation, in which case blocks having sidedimensions on the order of 3-7% those of the watermark signal image canbe used. Best, from an edge artifacts standpoint, are style images withvery little autocorrelation, in which blocks having side dimensions onthe order of 7-20% those of the watermark signal can be used. This lastsize produces the smallest number of edge continuities within thecollage. (Put another way, style images with the highest translationentropy are generally preferred.)

FIG. 4 shows, on the left, a watermark signal tile that serves as atarget image, and a coffee bean image that serves as a style image,which are processed to produce the collage shown on the right.

It will be recognized that there is a great deal of spatial repetitionin the coffee bean pattern. That is, its autocorrelation is high. Thisrequires small block sizes. One block is outlined in white in thecollage image. The arrows point to the edge boundaries between adjoiningblocks in the collage image—boundaries that can be discerned withcareful inspection. (The horizontal edge boundaries are not particularlyidentified.) In this composite image there are 64 blocks. That is, eachblock occupies about 1.6% of the watermark signal image area.

FIG. 5 conceptually shows the process. A regular array of blocks isidentified within a watermark signal image. For each watermark signalblock, a corresponding block is identified from within a style image,based on a similarity measure such as correlation, and copied to thatposition in the watermark signal image (or into a corresponding outputimage frame).

In a particular embodiment, the watermark signal image is divided intoan array of k×k pixel blocks. Each of these is correlated against everyk×k pixel excerpt within an N×M pixel style image, by across-correlation operation. For a k×k watermark signal block X, thecross-correlation operation with a single same-sized block Y in thestyle image can be expressed as:

${{xcorr}\left( {X,Y} \right)} = {\sum\limits_{j = 1}^{k}{\sum\limits_{i = 1}^{k}{\left( {x_{ij} - m_{x}} \right)\left( {y_{ij} - m_{y}} \right)}}}$Where m_(x) is the average value of the watermark signal block X, andm_(y) is the average value of the style image block Y.

This double-summation expression can be rewritten as a matrix functionB, as indicated by FIG. 6. Each column in this matrix is the vector ofk² pixels y=[y_(ij)−m_(y)] in each k×k block of the style image, wherem_(y) is the mean value of the pixels in that block. There are(M−k)(N−k) blocks of size k×k in the style image of size MN. Hence weobtain a matrix of size k²×(M−k)(N−k) by stacking all the vectors y ascolumns. (For large style images, we may prune the number of columns inthe matrix B, by choosing a random sub sample of the columns up to atarget number of columns.) Similarly, we create a vectorx=[x_(ij)−m_(x)] from the image block X in the watermark signal image.

From this matrix expression we may write:xcorr(X,Y)=x ^(T) ya simple vector inner product operation.

Then the co-ordinates u, v of the block in the style image with the bestcorrelation with the current watermark signal image block, out of allpossible blocks in the style image, are given by:

$\max\limits_{u,v}{x^{T}B}$

The edge artifacts caused by tiling disparate blocks from the styleimage can be mitigated by considering edge continuity as a factor whenselecting blocks from the style image.

The arrows in FIG. 7 show a progression of adding blocks to the collage.To reduce edge discontinuities, an edge discontinuity metric (orsimilarity metric) can be employed. One may be termed a Total Variation(T.V.) metric and is defined as:T.V.=Σ_(i=1) ^(k)(|b _(i) −t _(i) |+|r _(i) −l _(i)|).This factor evaluates congruence between the bottom edge (“b”) edge ofthe previously-added style block above, and the top-edge (“t”) of thestyle block under consideration, and similarly considers congruencebetween the right edge (“r”) of a previously-added style block to theleft, and the left edge (“1”) of the style block under consideration. Inone exemplary arrangement, the absolute differences in pixel valuesalong each of these two edges are summed.

(At the outer edges of the watermark signal image, where there is noadjoining block, we assume uniform pixel values of 127—mid-grey in a0-255 pixel scale.)

To reduce edge artifacts, we want to minimize this T.V. metric, which isthe same as maximizing its negative value. We thus combine the TV metricwith the cross-correlation metric in a weighted sum to create a singlemetric for optimization:

$\max\limits_{u,\nu}\left\lbrack {{\alpha x^{T}B} - {\beta\left( {\sum\limits_{i = 1}^{k}\left( {{{b_{i} - r_{i}}} + {{r_{i} - l_{i}}}} \right)} \right)}} \right\rbrack$where α is a weighting factor for the correlation term, and α is aweighting factor for the edge continuity term. Put another way,parameters α, β determine the signal strength and embedding smoothness,respectively.

The result does not change by multiplying both of the parameters by acommon factor; only their ratio matters. For example, to get the mostsignal, but also the most discontinuous edges, the parameters can be setas α=1, β=0. At the other extreme, with α=0,β=1, there is nil watermarksignal, but smooth edge transitions.

FIGS. 8A-8D illustrate. FIG. 8A shows a pattern of concentric circles,used as a style image. FIG. 8B shows a collage produced using excerptstaken from the concentric circles style image (and the watermark signalimage of FIG. 4), when the correlation metric α=1 and the TotalVariation metric β=0. FIG. 8C shows the opposite case, when edgesmoothness is exalted over signal strength (α=0, β=1). FIG. 8D shows anintermediate state, in which parameters α, β both have non-zero values.

(One might expect FIG. 8C to match FIG. 8A, since FIG. 8A has the fewestedge artifacts. The unexpected FIG. 8C pattern is due, in part, to theblock with which tiling started, and the fact that blocks are chosenbased on continuity along only two edges—since the other two edges don'tyet have blocks in place.)

Matlab code for producing the patterns of FIGS. 8B-8D is shown in FIG.9A, which continues onto FIG. 9B. (The im2col function divides an imageinto blocks, converts each block to a vector, and creates a matrix bylisting the vectors in columns.)

FIGS. 10, 11 and 12 show other examples of stylized watermark signalsproduced by the just-detailed arrangements—each with a style image onthe left, and a resulting collage image on the right. FIG. 11A issimilar to the collage shown on the right of FIG. 11, but thesynchronization marks of the original QR code used as a style image aremaintained, to reinforce the visual association with a QR code. Each ofthe collage images of FIGS. 10, 11, 11A and 12, like the collage imagesof FIG. 4, FIGS. 8B and 8D, readily reads using the Digimarc Discoverdigital watermark detector, available from the Apple and Android appstores.

The collage images of FIGS. 10, 11, 11A and 12 read best when presentedat a scale of about 100 waxels per inch or larger (e.g., 75 WPI). Atsuch scale, the depicted collages are at least 1.28 inches on a side.Below this size their reading becomes less robust. (On the other hand,below this size, their artiness become less visible—reducing the appealof using such stylized patterns.) Better robustness at smaller physicalsizes may be aided by applying a low frequency boost to the watermarkimage prior to the block matching process, and by employing smallersub-block sizes for the collage.

If the style image is in RGB color, one particular embodiment convertsthe style image to a counterpart luminance image (e.g., using a RGBweighting of 0.21 R+0.72 G+0.07 B) for matching. That is, blocks fromthe watermark signal image are checked for correlation and edgecongruence with greyscale blocks resulting from this conversionoperation, to determine locations, within the style image, of thebest-matching blocks. The original color image blocks from thethus-determined locations are then copied from the style image into theoutput frame to produce a color collage.

If the style image is in color, the discontinuity metric discussedearlier can be modified to also assess color discontinuity along theadjoining edges.

Applicant earlier referred to blocks having side lengths ranging from 1%to 20% of the watermark image side length. Blocks larger than 20% wouldtheoretically be better but are statistically impractical.

An exemplary watermark tile comprises 16,384 waxels, arrayed in 128columns and 128 rows. A block with side dimensions of 20% of such a tilehas 25 rows and 25 columns, or 625 elements. (This assumes eachwatermark element is represented by a single pixel. Commonly, a “bump”comprising 4, 9 or 16 pixels is used.) Even with just two values, thereare 2 {circumflex over ( )}625 different 25×25 watermark blocks—adecimal number that is 188 digits in length. If the style image isregarded as a dictionary from which different blocks are taken tocompose the mosaic, an ideal such dictionary would have a similar numberof different blocks that could be excerpted. For a style image to have10{circumflex over ( )}188 different 25×25 excerpts, it would need tohave at least 10{circumflex over ( )}94 rows, and a like number ofcolumns (assuming no 25×25 excerpt is repeated in such an image).

The situation is somewhat better with block sides measuring 10% of thewatermark image sides. Such a block would be 13 pixels on a side, or 169elements. Since each element may have two different values, there are 2{circumflex over ( )}169 different such blocks—a decimal number with 50digits. For a style image to serve as a dictionary with this manydifferent blocks, it would need to have at least 10{circumflex over( )}25 rows, and a like number of columns.

Surprisingly, applicant found that image dictionaries (i.e., styleimages) of such sizes are not required. Due to data redundancies and theerror correcting capabilities of coding used in an exemplary watermark,much much smaller, and much much more practical, style images can beused. The examples discussed herein employ style and watermark imagesmeasuring 256×256 pixels. If a 10% block size is used, each block isabout 26 pixels on a side. There are 52,900 such blocks that can beexcerpted from a 256×256 pixel style image (i.e., (256−26)*(256−26)),and this has been found to be generally adequate in most cases.

Roughly speaking, most typical applications of the technology (e.g.,when mimicking watermark images having a 128×128 array of waxels, usingblocks whose sides are at least 6% that of the watermark image)generally require a style image dictionary from which more than 10,000different blocks can be drawn. A minimum dictionary size of 20,000 isbetter, and 40,000 or more is still better. The larger the dictionary,the more faithfully the collage can mimic the watermark signal image,and the more reliably such a collage will be readable by a digitalwatermark reader.

However, as the side length of the block increases, the size of theneeded imagery dictionary increases exponentially. When the side lengthexceeds 20% of the watermark image dimension, there are typically notenough different style image blocks that can be excerpted to mimic thewatermark image with the degree of fidelity needed to assure reliablewatermark reading—at least with style images of practical size.

(At the other extreme, very small image dictionaries may be used. In thelimiting case of a block having a size of one pixel, a dictionary ofjust a few differently-valued pixels, e.g., 32 or less, can suffice.)

The field of Photomosaic art, developed by Silvers (c.f. U.S. Pat. No.6,137,498), seeks to mimic target artworks (e.g., famous portraits) inmosaic form, using other photos as mosaic elements. Such a mosaic,especially if seen from a distance or blurred (e.g., by a Gaussianfilter), immediately evokes—to human viewers—the original targetartwork. The present technology is different from Silvers' in a numberof respects—a primary difference is that the resulting mosaic does notappear, to human viewers, like the target watermark signal. Rather, theresulting mosaic evokes, to human viewers, a visual impression from thestyle image. Put another way, if viewers are asked whether applicant'smosaic bears more resemblance to the style image or to the watermarkimage, nine viewers out of ten (or more) will respond saying itresembles the style image. See, e.g., FIG. 4.

This difference is due, in large part, to Silvers' concern with humanperception, and applicant's concern with watermark decoding. Silverstries to replicate an RGB image with full fidelity in the color space,to immediately evoke the target artwork to human viewers. Applicant ismore concerned with faithfully reproducing pixel-scale gradients, andphases, in greyscale representation—the feature spaces on whichwatermark embedding and reading depend.

More on Edge Artifacts

Many different approaches can be employed to mitigate edge artifactsthat arise when blocks are mosaiced together to mimic a watermark tile.Some rely on selecting style blocks based, in part, on edgecongruence—an example of which is detailed above.

Other approaches can use weighting masks, in conjunction withoverlapping blocks. In a first such approach, shown in FIG. 13, boldgrid 131 indicates a first tiled array of blocks selected from a styleimage. A fine grid 132 indicates a second tiled array of blocks selectedfrom the style image.

The first grid 131 represents a complete 128×128 waxel watermark,composed of 16 blocks A-P, each mimicking a 32×32 waxel excerpt of thefull watermark signal. (A further row and column of 32×32 blocks areshown, as the watermark tile is designed to be continuous between itsleft and right edges, and likewise between its top and bottomedges—so-called spatial wrap-around, or toroidal wrapping.)

The second grid 132 also represents the same complete 128×128 watermark,albeit shifted right by 16 waxels and shifted down by 16 waxels. (Thatis, the upper left waxel in the fine grid 132 corresponds to waxel{16,16} in the watermark tile.) Again, grid 132 comprises blocksselected from a style image to mimic corresponding patches of thewatermark signal.

Thus, in this example, instead of selecting 16 blocks from the styleimage to mimic corresponding patches of the watermark tile, 32 blocksare selected.

To blend these two overlapping arrays of blocks together, a 2D weightingfunction, or mask, is applied to each of the 32 blocks selected from thestyle image.

The 2D weighting function has the property that, if a block is shiftedby half its side (say L), the sum of the original and shifted functionsis a constant. That is, for each point in the 2D space, the weightsapplied to the two overlapping pixels sum to a constant.

There are many choices for this function. The following function isexemplary:

${F\left( {x,y} \right)} = {{\sin\left( \frac{\pi x}{L} \right)}^{2} + {\sin\left( \frac{\pi y}{L} \right)}^{2}}$In this case, the weighting function, offset by L/2, is given by

${F\left( {{x + \frac{L}{2}},{y + \frac{L}{2}}} \right)} = {{{\sin\left( \frac{\pi\left( {x + \frac{L}{2}} \right)}{L} \right)}^{2} + {\sin\left( \frac{\pi\left( {y + \frac{L}{2}} \right)}{L} \right)}^{2}} = {{{\sin\left( {\frac{\pi x}{L} + \frac{\pi}{2}} \right)}^{2} + {\sin\left( {\frac{\pi y}{L} + \frac{\pi}{2}} \right)}^{2}} = {{\cos\left( \frac{\pi x}{L} \right)}^{2} + {\cos\left( \frac{\pi y}{L} \right)}^{2}}}}$From the trigonometric identity sin(x)²+cos(x)²=1, it can be seen thatthis weighting function F has the desired property. That is:

${{F\left( {x,y} \right)} + {F\left( {{x + \frac{L}{2}},\ {y + \frac{L}{2}}} \right)}} = {2\mspace{14mu}{for}\mspace{14mu}{all}\mspace{14mu} x\mspace{14mu}{and}\mspace{14mu}{y.}}$

FIG. 14A graphically shows this weighting function, F, that can beapplied to each of the 32 blocks selected from the style image. Thelightest, central, region indicates a weight of 100%. The darkest,corner, regions indicate a weight of 0%.

It will be seen that, if the weighting function of FIG. 14A is appliedto selected blocks A, B, E and F in FIG. 13, the composite functionsweight these blocks as shown in FIG. 14B. The dashed square printed inthe center of FIG. 14B indicates where a block Q from the second grid132 of blocks is overlaid. It will be appreciated that if block Q isalso weighted by the same function F (i.e., FIG. 14A), and summed withthe outlined weighting function from blocks A, B, E and F (FIG. 14B),the weights will sum to a constant, as shown by the summation of FIG.14C.

The weighting function F thus does not serve to make any region of thefinished collage lighter or darker than any other region, since everyregion is composed of two weighted contributions (i.e., from the firstand second grids of blocks), and the weights applied to the twocontributions always sum to a constant. (At outer edges of the mosaic, aslightly different weighting function can be applied—one that maintainsa 100% weight to the outer edge.) Yet it will also be seen that alongthe four edges of the FIG. 14A weighting function F, the weights arediminished in amplitude, even diminishing to zero in the four corners.This serves to diminish edge discontinuities where, e.g., blocks A and Badjoin. In such regions, the overlaid block from the second grid (e.g.,block Q) is more dominantly weighted. Points falling along the proximateboundaries of blocks A, B, E and F fall within the interior of block Q,and these interior (and continuous) features from block Q then dominatethe sum. Edge discontinuities are thereby greatly reduced.

(Since the constant value of the weighting function sum is 2, values ofpixels in the summed result are divided by two to yield the finalresult.)

FIGS. 15A and 15B show two arrays of blocks selected from the styleimage of FIG. 8A, corresponding to the first and second block arrays131, 132 of FIG. 13, after each block has been weighted by the functionF of FIG. 14A. FIG. 15C shows their summation (after division by 2). Ascan be seen, there are no apparent edges where one block adjoins thenext.

FIG. 16B shows application of this method to produce a watermarkedcollage using blocks copied from the water droplet style image of FIG.16A to mimic a watermark signal. Likewise, FIG. 17B shows a watermarkedcollage produced using blocks copied from the style image of FIG. 17A tomimic a watermark signal. As is evident, none of FIG. 16B or 17B showsedge artifacts.

FIGS. 18A, 18B and 18C show Matlab code used with the just-describedarrangement.

A second, variant, weighted-overlay approach uses the weighting functionshown in FIG. 19A, where white indicates a weight of 1.0, and blackindicates a weight of 0. If such a function is applied to each of blocksA, B, E and F of FIG. 13, the pattern of FIG. 19B results. The dashedsquare again shows where block Q, from the second grid of blocks, isoverlaid. Again, if block Q is weighted by the same pattern, the summedresults again yield a constant value at all positions, as shown by FIG.19C. (Again, at the outer edges of the collage, no summation orweighting is needed—those blocks can extend all the way to the outeredges with a weighting of 1.0.)

No division by two is required with the FIG. 19A weighting function,since one function always has a value of 1.0 where the other has a valueof 0, and vice versa.

The FIG. 19A weighting function is advantageous in certain respects,relative to the FIG. 14A weighting function. For example, when using theFIG. 14A weighting function, most points in the collage reflect non-zerocontributions from two different blocks from the style image. This canproduce ghosting or blurring effects where both components are evident,as is visible from close inspection of FIG. 15C. Relatedly, the bitonalblack/white aspect of the FIG. 8A style image is lost; the finished FIG.15C pattern including grayscale tones.

Additionally, the superposition of two staggered collages can result ina doubling of features. This doubling, or busyness, is conspicuous inFIG. 16B, in the increased number of raindrops, and likewise in theincreased number of paper wrinkles in FIG. 17B. (However, in othersituations, this attribute is not so evident.)

The FIG. 19A weighting function is disadvantageous in other respects.Notably, pairs of overlaid blocks have edge-transitions along theinclined lines of the weighting function of FIG. 19A. An edgediscontinuity metric can be used to select blocks that yield lessconspicuous artifacts along such boundaries. Nonetheless, edgetransitions associated with the FIG. 19A weighting function aregenerally preferable to those found in the first-described arrangement(e.g., FIG. 7) since such transitions zig-zag through the collage, andare not as conspicuous as parallel sets of vertical and horizontalboundary lines traversing the pattern.

This highlights a further advantage of the FIG. 14A weighting function:no consideration of edge matching is required. Blocks can be selectedwholly on the basis of their correlation with the target watermarkpattern. The value δ in the earlier equation can be set to zero.

A third variant approach to avoiding edge artifacts, using overlappingblocks, is shown in FIGS. 20A-20C. In this approach, a band of pixelsaround the edge of each selected block is weighted in a linearvariation, transitioning from 100% nearer the center of the block (shownin white), to near 0% at the outer edges of the block (shown in black).This tapered weighting is positioned to overlie a complementary taperedweighting of the adjoining block, so that the overlapping weights sum to100%

This arrangement requires that the blocks selected from the style imagebe larger than the corresponding blocks of the watermark tile. Forexample, if the watermark tile is divided into 30×30 pixel blocks, and ataper-weighted border of 4 pixels is utilized, then the blocks selectedfrom the style image are padded to be 34×34 pixels in size. Theseselected blocks overlap the adjoining block by 4 pixels along each edge:2 inside the boundaries of the corresponding watermark tile block (shownin dashed lines in FIGS. 20A and 20B), and 2 outside. Exemplaryweightings for the four overlapping columns of pixels between blocks Aand B in FIGS. 20A-20C are shown in the following table:

Block A, cols. 5-30: ### weight = 100% Block A, col. 31: weight = 80%Block B, col. 1: weight = 20% Block A, col. 32: weight = 60% Block B,col. 2: weight = 40% Block A, col. 33: weight = 40% Block B, col. 3:weight = 60% Block A, col. 34: weight = 20% Block B, col. 4: weight =80% ### Block B, cols. 5-30: weight = 100%The area of overlap between blocks A and B, where paired weightstransition from favoring one block to the next, summing to 100%, areshown by the parallel lines in FIG. 20C. (Of course, transition borderzones larger or smaller than 4 pixels in width can naturally be used.Corners are also treated differently, in a straightforward manner—suchas by dividing the sum of overlaid pixel values by the sum of theirrespective weights.)

In this arrangement, a discontinuity metric may be employed to helpselect blocks from the style pattern that will help further avoid anyvisual edge artifacts.

Still other variants overlap selected blocks with gaussian weights—witha maximum weight at the block center, and tapering towards the edges.Overlapping pixels from two or more blocks can be averaged in accordancewith their respective weights.

Collages that Mimic a Previously-Watermarked Image

The discussion to this point has focused on collages that mimic thestatistics of a pure watermark signal tile, e.g., as shown in FIG. 4.The technology can likewise be used to mimic a previously-watermarkedgraphic artwork. FIG. 21 shows an example.

FIG. 21 shows, on the left, a denim “style” image and apreviously-watermarked “target” image (Lena). Blocks from the former aremosaiced, as described earlier, to yield a collage image mimicking thelatter.

Although the watermark is not immediately apparent in the FIG. 21 targetimage, it is present—it is simply encoded at a sufficiently-lowamplitude (and scaled in accordance with human visual systemsensitivity) so as to be easily-overlooked by humans. But if presentedto a watermark reader, such as the Digimarc Discover reader citedearlier, it decodes (as does the collage).

Unlike the Silvers photomosaics discussed earlier, the collage of FIG.21 evokes twin visual impressions in the observer. One is that of thetarget image. And one is that of the style image.

The visual impression of the target image within the resulting collegeof FIG. 21 is relatively subdued. Prominence of the target image withinthe resulting collage can be increased, without impairing watermarkreadability, by changing the average luminance of each block of thecollage. (Changing the average block luminance changes the low frequencycontent of the image. Watermark readability, in contrast, depends on thehigh frequency content of the image.)

More particularly, the average luminance of each block in the collagemay be adjusted to reduce its dependence on the luminance of theselected block copied from the style image, and to increase its valuebased on the luminance of the corresponding block of the target image.

In a particular embodiment, each block selected from the style image(StyleBlock) is processed using a weighting factor w to yield anadjusted style block (StyleBlock′), as follows:StyleBlock′=StyleBlock−w*mean(StyleBlock)+w*mean(TargetBlock)(The two weights w applied in this equation are typically the same, butneed not be.)

Thus, a w weight of 0 corresponds to the mean coming from the styleimage only and no contribution from the target image. At the otherextreme, a weight of 1.0 corresponds to the mean coming from the targetimage only.

FIGS. 22A-D show collage images resulting from such processing.

FIG. 22A shows the result when w=0. This is the original case, depictedin FIG. 21, in which no adjustment is made to average block luminances.The mean of each block in the collage is simply the mean of each styleimage block from which the collage is assembled.

FIG. 22B shows the result when w=0.2. FIG. 22C shows the result whenw=0.5. FIG. 22D shows the result when w=1.0.

As is evident, best results are generally achieved for w values between0.2 and 0.5. Below w=0.2, expression of the target image is subdued, toa commonly-undesirable degree. Above w=0.5, the mean values of thetarget blocks tend to dominate, yielding a gross pixelated aspect to theresulting collage image. Colors of the style image also tend to beincreasingly-overwhelmed by the greyscale offset of the added luminance,at high values of w.

Another approach to emphasizing the target image is to remove, from eachstyle block selected for the collage, a weighted portion of its averageluminance, and then add-in a weighted portion of the target image itselfto the resultant collage to change its luminance accordingly.

That is, each selected style block (StyleBlock) is processed to yield anadjusted style block (StyleBlock′), as follows:StyleBlock′=StyleBlock−w*mean(StyleBlock)And then the collage (Collage) is adjusted to yield an adjusted collage(Collage′), as follows:Collage′=Collage+w*(TargetImage)(Again, the two weights applied in these equations are typically thesame but need not be.)

FIGS. 23A-23D show results of such approach, when the Lena image of FIG.21 is stylized using an image of grass. FIG. 23A shows the result when aweight w of 0 is applied (i.e., no adjustment of style blocks nor thecollage). FIGS. 23B and 23C show the results when weights w of 0.2 and0.5, respectively, are applied. FIG. 23D shows the result when a weightw of 1.0 is applied. Again, weights in of about 0.2 to 0.5 are againusually preferred.

In addition to avoiding the pixilation/blockiness associated with high-wweights in the former approach (e.g., as in FIG. 22D), this approachalso reinforces the watermark signal in the resulting collage. (Sincethe target image is watermarked, adding a weighted fraction of thatimage into the collage adds watermark signal back in also.)

It should be understood that there are many edgesimilarity/discontinuity metrics that can be employed; the detailed onesare exemplary only. In some arrangements, similarity is determined usinga metric based not on pixel values, but on divergences of pixel valuesfrom a local mean (e.g., a criss-cross kernel or 3×3 kernel). Edgediscontinuity may be based on a sum of squared differences measure,instead of on a sum of differences measure. Other edge measures takehuman visual system factors into account, factoring-in spatial-,contrast-, and color-sensitivity functions. Many other variants will beapparent to the artisan.

The RMS block similarity measure employed by Silvers may be used withthe present technology, in greyscale, if a large enough dictionary ofdifferent style image blocks is available.

In still further embodiments, block matching can employ the phasedeviation approach detailed in applicant's U.S. Pat. Nos. 9,959,587 and10,242,434.

The dictionary of image blocks available from the style image can beenlarged by considering the style image at different angular rotations,e.g., 90, 180 and 270 degrees—as well as at intermediate angles (withinterpolation). Multiple style images can be employed, although in suchsituations is it typical for all to depict the same subject matter. Anexample is to take a style image and scale it by 3%. The scaled imagewill provide excerpts with a new, different, set of statistics, whichmay provide better matches with certain blocks from the watermark image.

As is evident from FIG. 8D, certain pixel patches from the style imagemay be used multiple times within a single collage. (See, e.g., themultiple uses of the bulls-eye copied from the center of the styleimage.)

In the foregoing discussion, applicant refers to the patches of imageryselected from the style image as “blocks.” However, applicant tends torefer to a full expression of a watermark signal as a “block,” as well.Plural of the former blocks are mosaiced to mimic one of the latterblocks. By being alert to this issue, confusion can be avoided. (Thefull expression of a watermark signal is sometimes referenced as a“tile” as well, which lead to other confusion.)

While the detailed arrangements employ square blocks, this is notessential. For example, the excerpts matched between the target imageand the style image may be rectangular in shape, e.g., patches that are8×16 (or 8×32) pixels in size. More generally, shapes other thanrectangles can be used; image excerpts of countless shapes can becollaged together. Triangles and hexagons are simple examples, but anyshapes that can be arrayed together to span the target image can beemployed—including tessellation tiling patterns composed of two or moredifferent shapes and/or sizes. Many such tiling patterns are detailed inthe appendices.

In addition to the noted Silvers patent, the following patents describeother photo mosaicking arrangements. The present technology can beincorporated into these systems, and vice versa: U.S. Pat. Nos.6,532,312, 6,665,451, 6,687,419, 6,724,946, 8,340,423, 20030001858,20040022453, 20040047518, 20050047651, 20050147322, 20110050723,20110058736, 20110103683, 20110268369 and 20150262335. For example,certain of these references teach methods of determining blocksimilarity, and dealing with edge artifacts, that can be employed in thepresent arrangements.

Section II: Artwork Generated by Code Transformations

In accordance with a different aspect of the technology, code-conveyingartwork suitable for use as a visible, aesthetic design element isproduced by applying one or more geometric transformations to the 2Dsignal code, imparting an organized structure to the code.

Some of the transformations that can be employed are shown in FIGS.24A-24H. FIG. 24A shows a proxy for an input image, prior totransformation. FIG. 24B shows this image transformed by rotation. (A 90degree rotation is shown, but any rotations can be used.) FIG. 24C showsmirroring of the input image. (Horizontal mirroring is shown, butmirroring around any axes can be used.) FIG. 24D shows scaling. FIG. 24Eshows inversion (light for dark) of the input image. FIG. 24F showstranslation (shifting) with wrap-around, of the input image. (Horizontalshifting is shown, but shifting in other directions can be employed aswell.) FIG. 24G shows an FFT-shift operation of the input image. (FFTshift is a half a block horizontal translation and half a block verticaltranslation, with wrap-around. It is a special case of a toroidal shift.The generalized toroid shift (or wrap) is not limited to shifts of ahalf block.) FIG. 24H shows a second input image. FIG. 24I shows thissecond input image after a glide-reflection transform, with wrap-around.(Reflection around a horizontal axis is shown, but reflection aroundother axes can be used as well.)

This is just a sampling of available transforms. Some of these (e.g.,FIGS. 24B, 24C, 24F and 24I) are members of the “wallpaper group” oftiling symmetries. The present technology can be used with any member ofthe wallpaper group. (The Wikipedia article for Wallpaper Group isattached as Appendix A to application 62/972,522 and forms part of thisspecification.)

All such transforms can be applied to parts, or the entirety, of aninput 2D signal code 250, as shown in FIG. 25. This code is producedusing methods known in the prior art and reviewed elsewhere herein(e.g., as in FIG. 1B). The code signal comprises a 128×128 array ofwaxels. Each waxel is represented by a 4×4 block of pixels. The codethus measures 512×512 pixels. The code may convey a plural-symbolpayload, e.g., of 50 bits.

FIG. 26 illustrates one illustrative transformation of this input signalcode 250. The code signal block is first divided into an 8×8 array ofsub-blocks (as indicated by the white lines, which are provided simplyto aid explanation, and have no physical counterpart in the signalcode). Each sub-block thus comprises 64×64 pixels. The FIG. 26 patternis produced by summing each original 64×64 pixel sub-block, with thatsub-block rotated by 90 degrees, and that sub-block rotated by 180degrees, and that sub-block rotated by 270 degrees.

For example, the upper left pixel in original signal code 250 is summedwith the pixel that is at row/column coordinates (1,64) in the originalsignal code, together with the pixel that is at (64,64) in the originalsignal code, as well as with the pixel that is at (64,1) in the originalsignal code. This sum of four pixel values is divided by 4 to normalizeit back to fall within the original range of pixel values (e.g., 0 to255). In like fashion, each pixel shown in transformed sub-block 261 isthe normalized sum of four pixel values, produced by rotating thecorresponding pixel sub-block in the original code signal by 0, 90, 180and 270 degrees.

The mathematical transform used in FIG. 26 is indicated by the Matlabequation shown at the top of the figure, where “x” is the original codesignal. “R4” is the name given by applicant to this pattern, indicatingeach sub-block is the sum of 4 rotated versions of the originalsub-block (with one of the rotations being 0).

FIG. 26A indicates, graphically, the normalized sum of the fourrotations of the upper-left sub-block from FIG. 25, which yields theupper left sub-block 261 of FIG. 26.

As can be seen from FIG. 26, each sub-block produced by this method ischaracterized by 4-way geometric symmetry, imparting a pleasingaesthetic to the resulting pattern. Each such processed sub-block may beregarded as a tile in a tiled pattern.

Despite such transformations, the FIG. 26 pattern still conveys theoriginally-encoded information to a watermark reader. Indeed, itsreadability is enhanced by the transformation. In particular, when awatermark reader is presented with captured imagery depicting the FIG.26 pattern (e.g., an image of a food package bearing this pattern,captured as the package is swiped over a glass platen of a point-of-salescanner), the reader does not need to consider all possible rotationstates with which the pattern might be depicted in the captured imageryas it tries to geometrically synchronize to the pattern (e.g., todetermine its scale, rotation, and the position of the upper left handcorner within the captured imagery, so that the encoded information canbe extracted). Instead, the reader can search just a 90 degree range ofrotations; the indicia is 90 degree rotationally-invariant. That is, thesymmetry induced by the just-detailed transformation enables the readerto synchronize to the pattern as depicted in the capturedimagery—regardless of its presentation in the captured imagery—byconsidering just a 90 degree range of possible rotations. This symmetryallows the watermark reader to synchronize to the pattern of FIG. 26more quickly, and/or with less computational resources, than it requiresto synchronize to the pattern of FIG. 25, when these patterns arepresented in captured imagery at unknown rotational states.

FIG. 27 illustrates another pattern resulting from transformationsapplied to the input 2D code signal. In this case each original codesignal sub-block is mirrored (flipped) left-for right, and summed withitself, to yield an interim product. This interim product is thenmirrored top-for-bottom and summed with itself, yielding the patternshown in FIG. 27. Again, since each pixel in the transformed sub-blockis based on a sum of four processed pixels (i.e., each of the four termsin the equation atop the figure), the sum is divided by 4 to normalize.(The “M4” in the Matlab equation atop the figure is applicant's way ofdenoting a 4-way mirroring transform.) Again, each sub-block in theinput image is processed to yield a structured tile.

FIG. 28 illustrates still another pattern resulting from transformationsapplied to sub-blocks of the input 2D code signal. In this case eachsub-block is shifted horizontally by one-quarter of its width, and byone-half of its width, and by three-quarters of its width. The fourvariations on the sub-block (i.e., the original and the three shiftedversions) are summed, and the sum is normalized, to produce one of thetiles of the pattern. The “T4” in the Matlab equation atop the figure isapplicant's way of denoting a transformation based on four suchtranslations.

Again, the FIG. 28 pattern eases the decoder's work. The decoder needsto search for a smaller range of possible translations, when attemptingto synchronize to an input code signal of unknown translation.

FIG. 29 shows yet another processed indicia. In this pattern eachsub-block has been transformed into a tile comprising what appears as a2×2 pattern of sub-tiles (corresponding white lines have been added);the entire block is comprised of a 16×16 pattern of such smallersub-tiles.

Each tile in this case is produced by an M4 mirroring operation appliedto each sub-block (detailed above and shown in FIG. 27) to yield aninterim-processed sub-block, which is then summed with an FFT-shiftedcounterpart of that interim-processed sub-block.

FIG. 30 shows still another such processed pattern, employing rotation,negation (inversion), and FFT-shifting. The input pattern x is summedwith the input pattern rotated 180 degrees, and from this sum aresubtracted the input pattern rotated 90 degrees, and the input patternrotated 270 degrees. This yields an interim pattern, which is thenFFT-shifted and added to itself.

The subtraction is the negation operation, causing each tile to includetwo inverted counterparts of the original sub-block. The presence ofsuch inverted counterparts allows the payload to be recovered fromcircumstances in which the light/dark pattern is inverted—as can happenwhen reading texture patterns from specular plastic surfaces (as notedin published application 20190306385, which is incorporated herein byreference in its entirety). The rotation operation again limits thesearch space of possibly-transformed counterparts that the detector mustexplore in finding correct synchronization to decode the payload fromthe pattern.

(The terms “sum,” “summation” and the like, as used herein, encompassoperations in which one or more of the operands may be negative. Thus,subtraction is a form of summation.)

FIG. 31 shows a pattern in which each component tile is produced by4-way rotation/summing of a sub-block (as in FIG. 26), and the resultinginterim tiles are then 4-way mirrored (as in FIG. 27).

Although the axes in the foregoing examples all pass through the centerof a sub-block, this is not essential.

In the above examples, a 512×512 pixel signal block is divided into anintegral number of sub-blocks. That is, the sub-blocks are sized toperfectly span the block (e.g., 64×64 pixels). In such arrangement, ifthe post-processed blocks are placed edge to edge (e.g., to span an areaof product artwork), the sequence of different tiles cyclically repeats.In FIG. 32, where two of the patterned blocks of FIG. 31 are placedside-by-side, the upper-left tile 321 of one block is identical to theupper-left tile 322 of a next block, etc.

A different result can be achieved by setting the sub-block size so thatit is not an integral fraction of the whole signal block. FIG. 33, forexample, shows tiles resulting from sub-blocks of size 72×72 pixels.7.111 such tiles span a 512×512 signal block (shown by the solid linesquare). In this case the tiles take different appearances, and do notrepeat from one block to the adjacent block. For example, the tile 331found in the upper left corner of one block is visibly different thanthe tile 332 found in the upper left corner of the adjoining block.(FIG. 33 is magnified relative to FIG. 32 to show additional detail.)

It will be recognized that the tiles of FIG. 33 are black and white, orbitonal, whereas the earlier examples were greyscale. A bitonal tile (orblock) can be derived from a greyscale tile by thresholding: setting allpixels of a particular value (e.g., below 128 or 96, or thelowest-valued 30% of pixels) to black, while setting other pixels towhite. Despite such further transformation to the original signalelements, the pattern of FIG. 33 still reads with a conventionalwatermark detector (e.g., the Digimarc Discover iPhone app).

In other arrangements the input watermark block can be divided intohexagonal sub-blocks, rather than squares. Such sub-blocks lendthemselves to mirroring, rotation, and other transformations along axesoriented, e.g., at 0, 120 and 240 degrees.

FIG. 34 shows an example employing six rotations, thus “R6.” In thiscase each sub-block is rotated to six different states (oriented at 0,60, 120, 180, 240, and 300 degrees). These differently-rotatedsub-blocks are cropped, summed, and the results normalized by dividingby six. Put in Matlab terms, this transform is:R6=@(x)(x+imrotate(x,60,‘crop’)+imrotate(x,120,‘crop’)+imrotate(x,180,‘crop’)+imrotate(x,240,‘crop’)+imrotate(x,300,‘crop’))

FIG. 35 shows another example: “M6.” In this case it is sufficient toperform the mirror/sum operation once (e.g., across a horizontal axisthat bisects each sub-block), and then sum counterparts of thisintermediate pattern, rotated at 0, 120 and 240 degrees. That is:

M=@(x) (x+flipud(x));

M6=@(x) (M(x)+imrotate(M(x),120,‘crop’)+imrotate(M(x),240,‘crop’));

(Normalization is omitted in these formulas.)

FIGS. 36A-36D show a further pattern that is produced by cascadingdifferent mirroring and rotation transformations on each sub-block in a2D signal block. In these four cases, all of the transformations producethe same output pattern.

FIG. 37 shows a nearly-circular pattern resulting from taking the full,square signal code block of FIG. 25, rotating it to 14 angular rotations(equally-spaced, here 12.86 degrees), summing, and normalizing (i.e.,scaling by an amplitude factor of 1/14).

FIG. 38 shows another variant in which the full code block of FIG. 25 isrotated to 60 equally-spaced angular rotations (i.e., 6 degrees apart),summed, and normalized. The full code block is also rotated to 10equally-spaced angular rotations (i.e., 36 degrees apart), summed, andnormalized. The results from these two sets of operations are thensummed and normalized.

As with some of the previously-discussed arrangements, thetransformations of FIGS. 34-38 enable a decoder to synchronize to acounterpart of the encoded information without searching the entirerange of possible rotation states. Instead, geometric synchronizationwill be found in a much-abbreviated search space.

FIG. 39 shows a further variant in which the full code block of FIG. 25is rotated to 60 equally-spaced angular rotations, summed andnormalized. However, at each successive rotation state the block isscaled by 104%, so the summation includes patterns at a great variety ofdifferent scale states. This extends the range of distances over which adecoder can successfully extract the encoded payload information. Whenthe decoder is at large reading distances, the pattern appearsrelatively small, but the largest-scale counterparts of the blocks stillfall within the decoder's range of permitted scale states, and thus arereadable. Conversely, when the decoder is at close reading distances thepattern appears relatively large, but the smallest-scale counterparts ofthe blocks still fall within the decoder's operational range.

FIG. 40 shows the pattern resulting from the FIG. 39 transformations,but employing 10,000 equally-spaced angular rotations.

FIG. 41 shows a pattern resulting from scaling a pattern at 21 differentscale states (each 95% the size of the previous one), summing andnormalizing. This pattern is readable over a range of reading distancesof about 25:1 using a standard watermark decoder (e.g., as provided inthe Digimarc Discover iPhone app)—about 10 times larger than thedecoder's normal reading range.

FIG. 42 shows another pattern resulting from both repeated scaling androtation transforms. This pattern is comprised of 20 equally-spacedrotations of the signal block, with each successively scaled by a factorof 0.98.

The arrangements described above process the entirety of the signalblock (in whole, or in sub-blocks) to yield the resulting pattern. Butthis is not necessary. Parts of the signal block, or of each sub-block,can be discarded, and a pattern can be formed by one or more transformsapplied to remaining elements.

A useful analogy is a kaleidoscope—an instrument in which two or moremirrors are tilted towards each other at an angle, causing a set ofitems positioned within this angle to be apparently replicated aroundaxes when viewed from an eyepiece. In like fashion, a spatial excerpt ofelements within a block or sub-block can be projected around one or moreaxes to generate symmetrical patterns.

An example is shown in FIG. 43A. In this case a sub-block is representedby the ABCD pattern of FIG. 24A. A vertical axis bisects the sub-block,and elements to the right of the axis are discarded. The remainingelements are mirrored around the axis to produce the pattern of FIG.43A.

A further example is shown in FIG. 43B. Here both vertical andhorizontal axes bisect the sub-block, and elements except those in theupper right quadrant are discarded. When reflected (mirrored) around thetwo axes, the pattern of FIG. 43B results.

A still further example is shown in FIG. 43C. Here vertical, horizontal,and diagonal axes define a one-eighth sliver. Elements except in thisone eighth sliver are discarded. When the remaining elements arereflected around the axes, the pattern of FIG. 43C results.

The discarding of some elements in a sub-block (or block), withpreservation of other elements in the sub-block, may be conceptualizedas a windowing function. FIG. 44A shows the windowing function used inthe example of FIG. 43A, with the retained area shown in cross-hatching,and the discarded elements shown by white. FIG. 44B shows the windowingfunction used in the example of FIG. 43B. FIG. 44C shows the one-eighthsliver identified by the window used in the example of FIG. 43C.

The discarding of signal elements by windowing does not prevent readingof the encoded information due to the highly redundant error-correctionencoding used in most forms of applicant's digital watermarking.(Particular examples are detailed in our patent publication 20190332840,which is incorporated herein by reference in its entirety.)

In the just-detailed arrangements, no normalization (e.g., division byfour) is required, due to the reflection of elements into otherwiseblank areas.

In other arrangements, normalization is required. Consider the inputsignal block represented by the Sample Design box of FIG. 45A. Awindowing function can be applied to select the left half of this block,and the selected half can be mirrored around the vertical axis, yieldingthe interim-processed block of FIG. 45B. No normalization is needed,since the selected half of the block was windowed onto a blank area.

The interim-processed block of FIG. 45B can then be divided into anarray of 16 sub-blocks. (Only three are shown, by the dotted lines.)Each of these sub-blocks can then be R4-processed to yield the finalblock shown in FIG. 45C (enlarged to show detail). Since the R4operation involves summing multiple counterparts of image data,normalization is here required. This normalization leads to thegreyscale tones in FIG. 45C, whereas the original signal block of FIG.45A is bitonal.

(As illustrated by the foregoing example, a windowing operation can beapplied before dividing into sub-blocks, rather than after—as in FIGS.43A-43C.)

When the just-described process is applied to the 2D signal block 250 ofFIG. 25, but dividing into an 8×8 array of sub-blocks, the pattern ofFIG. 45D results. (Again, white lines separating the processed tiles areadded to aid visual understanding.) A watermark detector can extract thepayload from this pattern without examining all possible rotations,since the use of the R4 operation constrains the search space that mustbe explored. This sequence of transformations also yields an array oftiles that can be read even if mirror-imaged, further enhancing itsutility.

(Windowing may be regarded as a sparsification operation—limiting theamount of input data that will be included in the output signal.)

FIG. 46 shows how a pattern produced by the foregoing methods can beprinted on the bottom of a food container (e.g., a yogurt tub). Printingmay be performed using ink-jet, offset, dry offset, or other technique.The encoded information may include a GTIN for the item, a lot code,expiration date information, etc.

As with other watermark patterns, patterns generated according to thepresent methods can also be combined with other artwork, in a ratiochosen to yield desired visibility and robustness. FIG. 47 is anexample. Although the Lena image is probably not suitable for productpackaging, it will be recognized that other images may be combined withsuch watermark patterns for use in packaging applications.

The patterning of tiles resulting from the foregoing methods becomesmore distinct as the sub-block size is reduced. FIG. 48A shows the M4transformation of FIG. 27 applied with sub-block sizes ranging from16×16 to 96×96. As can be seen, with larger sub-blocks, the patterningbecomes more nuanced and less overt. FIG. 48B shows a signal blockM4-processed using sub-block sizes of 128×128. FIG. 48C shows a signalblock M4-processed using sub-block size of 512×512 pixels. In thisinstance it will be recognized that the sub-block is the size of theentire input signal block. Again, the larger the sub-block size, themore subtle is the patterning introduced by the transformationoperations.

Applicant has found that the “kaleidoscopic” approach, in whichwindowing produces a blank area in to which other imagery is rotated,mirrored, translated, etc., seems to work best with large sub-blocksizes (e.g., of a fifth of a block or larger), while theearlier-detailed approach, in which all input data is preserved andtransformed, seems preferable with smaller sub-block sizes. Thus, thepatterns of FIGS. 48B and 48C would have more visible organizedstructure if windowing were employed, in a kaleidoscopic approach.

While the arrangements detailed in this section take, as input, a 2Dsignal block alone, this is not essential. Instead, the input block cancomprise artwork that has been previously-processed to convey a 2D codesignal. Examples include the collage images earlier-detailed in SectionI, as well as the stylized artworks detailed in published application20190213705.

FIG. 49 is a flow-chart of an exemplary process incorporating certainaspects of the present technology. A 2D machine readable code mayoptionally first be pre-processed. This can include operations such assparsification (e.g., as detailed in 20190332840), conversion to bitonalform, greyscale remapping, colorization, etc. It can further includeincorporation into other artwork elements, as just-mentioned.

The block is then sub-divided into sub-blocks. (As illustrated by FIG.48C, a sub-block can be the entire block.) A windowing function can nextbe applied, to identify particular sub-block excerpts that are to beoperated on. (Again, the excerpt can be the entire sub-block.) Atransformation function is then applied. The transformation functions ofFIGS. 24B-G, and I, are most commonly-used, although others can beemployed. These transformations often involve summingdifferently-processed counterparts of the windowed excerpts to yieldtiles that exhibit a organized structure—often geometrical symmetry—notpresent in the input code.

Although the methods so-far detailed have been applied to an entiresignal block, this is not required. Such methods can be applied to justan excerpt. FIG. 50 gives an example, in which a single sub-block from asignal block is processed and thresholded to give it avisually-appealing symmetrical structure, but in which the surroundingsignal is simply sparsified. (A distinctive pattern, like that shown inthe center of FIG. 50, can serve as a “call to action,” e.g., indicatingthat consumers should capture imagery of such area of a package withtheir smartphones to be linked to additional product information, suchas nutritional information.) The particular pattern shown in the centerof FIG. 50 is produced by an R4(M4) operation, and is found as a tile inthe lower right of FIG. 33.

Moreover, it is not necessary that the transforms be applied tosub-blocks of the same size or shape. Any sizes or shapes, orcombinations thereof, can be used, giving rise to countless tilingarrangements. A few examples are shown in FIGS. 51A-51E. In some casesthe tiling arrangements may span multiple full signal blocks. And notall tiles need be processed in the same manner. For example, some tilesin a pattern (e.g., the darker tiles in FIG. 50D) may be produced by anM4 transform, while other tiles in the pattern (e.g., the lighter tilesin FIG. 50D) may be produced by an R4 transform, etc.

(Further examples of repetitive patterns that can be used with thepresent technology are detailed in the appendices, which includeWikipedia articles for Wallpaper Group; Truncated Square Tiling; andList of Euclidean Uniform Tilings.)

Although reference has been made to M4, M6, R4 and R6 transforms, itwill be recognized that M2 and R2 transforms are also highly versatile,as they can be used—without cropping—with sub-block shapes having only asingle axis of symmetry. Naturally, rotations and mirroring transformsof still different orders can also be used.

Still other aesthetic patterns can be achieved by applying a geometricaltransformation to the 2D code signal as one of the pre-processingoperations in FIG. 48. Then, at the end of the process, aninverse-geometrical transformation is applied. Such a method is shown inFIG. 51. (The sub-dividing and windowing operations are reversedrelative to FIG. 48, as these operations can be performed in any order.)

One such geometrical transformation is asymmetrical scaling. Forexample, the signal block 250 of FIG. 25 may be stretched by a factor oftwo in the horizontal direction. It is then divided into an array ofsquare sub-blocks, as shown by the stretched signal block of FIG. 52.Each square sub-block is processed, e.g., by an R4 transformation, toproduce stylized square tiles. Then, the stretched signal block,comprised of the stylized tiles, is scaled back to its square shape byan inverse geometrical transformation—compressing it by a factor of twohorizontally. The resulting stylized tiles comprising the restoredsquare block are no longer square—they are rectangles, twice as tall asthey are wide.

Another geometrical transformation is a conformal mapping. Conformalmappings preserve local angles, while preserving rotational and mirrorsymmetries.

FIG. 54 illustrates an example using conformal mapping. An input signalblock is shown in the upper left. It is conformally mapped to producethe pattern shown in the upper right. The pattern shown in the upperright is then divided into sub-blocks (here hexagonal), which aretransformed as detailed above (here using an R6(M) transform, asdiscussed in connection with FIG. 36B), yielding the pattern shown inthe lower right. An inverse conformal mapping is applied to this patternto yield the output pattern shown in the lower left. Strange but true: astandard watermark detector is readily able to read the payload fromthis output pattern.

The conformal mapping used is this example is the function f(z)=1/z.This is probably the simplest conformal mapping, but there are manyothers—each of which can be used with the present technology. Some otherexamples include:

-   -   f(z)=z{circumflex over ( )}2/2    -   f(z)=1/(2z{circumflex over ( )}2)    -   f(z)=SQRT(2)*SQRT(z)    -   f(z)=e{circumflex over ( )}z    -   f(z)=sin(z)        Generally speaking, any polynomial function will work.

As is familiar to graphic artists, a greyscale image can be made lighteror darker by remapping the grey tones to different values. For example,the pattern of FIG. 36A can be tone-mapped to produce the lighterpattern of FIG. 55A.

Another way of lightening or darkening a pattern is by screening withanother pattern. One such example is shown in FIG. 55B. This is thepattern of FIG. 36A now overlaid by a pattern of white lines. (The webof white lines is made visible at the margins to aid visualunderstanding.) Again, due to the highly redundant encoding of thepayload in the underlying pattern, the loss of parts of such pattern dueto masking by the overlaid line does not prevent watermark reading.

By making the overlaid lines wider or narrower, the composite patterncan be made lighter or darker.

In this case the web of lines is a Voronoi pattern derived from a sparsedot pattern, which in turn is derived from a watermark pattern (e.g.,using the methods detailed in application Ser. No. 16/435,164, filedJun. 7, 2019, published as 20190378235, the disclosure of which isincorporated herein by reference). Thus, the artwork of FIG. 55B canconvey two payloads: one in the watermark of the hex-stylized artwork,and one in the watermark of the overlaid screen pattern. Alternatively,both watermarks may convey the same payload, in which case the twowatermarks are desirably overlaid in registered alignment, with the samescale, rotation and translation.

If the two watermarks convey different payloads, a watermark signaldetector can be configured to decode one or the other, e.g., by usingdifferent reference signals or spreading functions in the twowatermarks, and employing parameters in the detector that are tailoredto one or the other. Alternatively, high-pass or low-pass spatial imagefiltering can be employed to accentuate one of the two watermarkpatterns in data submitted to the decoder, causing that pattern to bepreferentially read.

Screens of different shapes and forms can naturally be applied, e.g.,regular or stochastic dot and line screens that define transparenciesand masks. Any pattern can be used as a screen.

Reference was made to converting a greyscale pattern to bitonal bythresholding. To make such a bitonal pattern lighter, the threshold canbe changed to reduce the amount of ink deposited. However, this mannerof adjustment can seriously compromise signal robustness. Applicant hasfound that thresholding to yield 50% ink coverage, followed by screeningas above-described, preserves more of the signal energy, for a givenlightness of the resulting pattern.

It will be understood that patterns produced according to this aspect ofthe present technology are characterized by an organized structure—onethat is usually apparent to human viewers of the pattern. When printedin an end-use application, the structure is typically stillhuman-visible—even if the pattern(s) is incorporated into other artwork(as in FIG. 47). That is, the organized structure patterns are commonlynon-steganographic in their end use.

Whether a tile has an organized structure can be determined in severalalternate ways. Often such structure manifests as visible symmetry,i.e., one or more axes of symmetry are present in a tile havingorganized structure. Sometimes such structure is indicated by a central“vortex” feature (e.g., as in FIGS. 37-39, 41 and 42), or by concentric,radial or spiral features (as respectively typified in FIGS. 39, 41 and42). Still further, tiles having an organized structure typically have ahigher auto-correlation that the input data from which they werederived. Furthermore, a histogram of pixel values within such a tilecommonly exhibits a “lumpiness” not found in the input data from whichit was derived. For example, the northeast quadrant of tile 271 in FIG.27 is comprised of pixels having values that are replicated in the otherthree quadrants. Thus, any pixel value found in the tile is found atleast four times. This is in contrast to the histogram of thecorresponding region of the input signal block, where some pixel valuesmay occur 1, 2 or 3 times. In such case the histogram of the tile hasfewer different pixel values, but no pixel value present in the tile ispresent less than four times. If a tile has any of the foregoingattributes, and the input data from which it was derived does not, thenthe tile can be regarded as having an organized structure.

While the foregoing discussion has details transformations in thespatial image domain, transformations can likewise be applied in thespatial frequency domain. For example, a 2D code indicia can betransformed from the pixel domain into a DCT representation. In the DCTdomain the representation can be sparsified by discarding elements. Forexample, the odd indexed DCT coefficients may be discarded, whileretaining the others. The processed DCT domain representation is thentransformed back into the spatial image domain to yield a structuredcode pattern. (As with other embodiments, thresholding can be applied inthe spatial image domain to convert the pattern to bitonal form, and tochange the pattern darkness.)

In contrast, it will be recognized that some transformationoperations—if applied singly—do not yield organized structure. Forexample, the structure of an input 2D code signal is not made organizedby just scaling it, or by just rotating it, or by just translating(shifting) it.

Moreover, applicant does not mean the present technology to extend toarrangements in which a steganographically-encoded watermark block issimply rendered at plural different scales, or other affine distortions,to extend a range of locations from which it can be read (e.g., astaught in our U.S. Pat. No. 8,412,577).

The following text is adapted from an appendix of priority application62/972,522.

The principles used to generate the signal carrying patterns are verygeneral and apply to any 2D signal which has the following ingredients.We assume the signal is a zero mean binary signal with black and whitepixels representing −1 and 1 respectively. So to have a zero mean, thenumber of black and white pixels are equal. We also define the signal asa square tile which may be tiled by repetition to cover a larger imageor cropped for a smaller image. An exemplary signal comprises (i) afixed zero mean synchronization signal, and (ii) a variable zero meandata signal derived from a 1D error correction code with high redundancy(on the order of 100 bits (pixels) per information bit). These twosignals are combined to produce a signal tile that may be seamlesslytiled to cover a larger image. The number of pixels in the signal tile,its physical size, the sync signal pattern, and the number ofinformation bits all vary in different applications of the technology.

In traditional watermarking, the goal is to embed a data carrying signalin any given cover image. FIG. 57 shows tradeoffs between the number ofsignal dimensions and the signal energy budget at a high level. Let'ssay we have 100 signal dimensions (pixels) to embed our signal. Intraditional watermarking we embed the signal at a fixed amplitude ineach dimension (this might differ in practice withadaptive/opportunistic embedding). In signal rich art we embed only(say) 10 signal dimensions, but use 10 times the signal energy in eachdimension, since the signal visibility is not a constraint. So theoverall energy budget (signal energy per information bit) is the same.However, there are options for choosing a subset of the total, whichgives an exponentially large space of signal carrying patterns to choosefrom. The signal recovery is also helped by the fact that we use anerror correction code with a high redundancy (>100×) per informationbit, so we can afford to drop 95% of the signal bits and still recoverthe data reliably from a noisy signal.

In the paper Kamath et al, Hiding in Plain Sight: Enabling the Vision ofSignal Rich Art, Electronic Imaging, Jan. 13 2019, pp. 527.1 527.8, weshowed how to generate signal carrying binary patterns by sparsifying awatermark signal. We here extend this idea to generate halftone dot andline patterns which can be used to generate signal carrying grayscaleimages.

FIG. 58 shows two classes of halftone patterns. The patterns on the leftare based on dots of a fixed spatial density (defined by the centers ofthe dots) and variable dot radii to achieve a grayscale variation. Thisstyle is also called amplitude modulation (AM) in the print industry.The line pattern counterparts have a fixed cell size (triangles forDelaunay and polygons/hexagons for Voronoi) and variable line widths. Onthe right we have dots of a fixed radius, but the density of the dotsvaries spatially to produce a grayscale variation. This is calledfrequency modulation (FM) in the print industry. The line patterncounterparts have a variable cell size and fixed line width in thiscase. To carry our signal, we choose the sparse dot locations based onlocal minima of the grayscale signal, which drives the pattern towardshigher correlation with the watermark signal. This increases thecomplexity of the halftoning algorithm, but allows us to embed thesignal along with the pattern as an integrated operation. FIGS. 59A and59B show excerpts of the halftone algorithm applied to a watermarkedgrayscale reindeer image with a Voronoi, FM pattern on the left and aDelaunay, AM pattern on the right. We had to lower the contrast of theimage on the left to achieve a more pleasing cell pattern and also forgood signal readability. We are able to maintain the signal strength andcontrast with the AM (variable line width) pattern.

CONCLUDING REMARKS

Having described and illustrated certain features of the technology withreference to illustrative embodiments, it should be apparent that theinvention is not so limited. Additional, complementary disclosure ofvarious embodiments is provided in Kamath, Ajith, Signal Rich Art:Improvements and Extensions, attached as Appendix B to application62/972,522. Appendix A and B are hereby incorporated by reference.

As noted, a particular application of the technology is in marking foodpackaging with watermarked identifiers, without the challenges ofkeeping the watermark pattern below a threshold of visual perceptibilitywhen rendered on a particular printing press. Instead, by creating apattern that is made plainly visible, the pattern serves as a deliberatecomponent of package artwork—one that contributes to, rather thandetracts from, package aesthetics. In this context, applicant has foundthat images of fruits, nuts, seeds, grains, beans, and other food items(e.g., gumballs), often serve as suitable style images for thearrangements detailed in Section I. Likewise, geometrical patterns areanother class of frequently-suitable watermark style images for foodpackaging, e.g., as detailed in Section II.

Naturally, the technology is applicable more broadly than in foodpackaging. Any application in which machine-readable information is tobe conveyed can utilize patterns created by the above-described methods.One such other application is in secure documents, where such patternscan be used as ornamental backgrounds or design elements for checks,passports, drivers licenses, airport boarding passes, and the like. Asin other applications, a reader device can decode a payload encoded bysuch a pattern, and take an action based thereon (e.g., opening aturnstile to admit a passenger to an airplane).

Although the technology is illustrated with reference to digitalwatermark technology, the principles are similarly applicable to othermachine-readable codes/symbologies.

The artisan is presumed to be familiar with methods of generatingwatermark blocks, e.g., from the patent documents cited herein. Withsuch knowledge by way of background, the process for generatingwatermark pattern of FIG. 1B can be summarized essentially as follows.An ensemble of 64 2D spatial sinusoids spanning the watermark block, ofdifferent frequencies and phases, are summed together and sampled (e.g.,based on a 128×128 waxel array) to form a reference signal component(used by the watermark detector to discern affine transformation of thewatermark pattern within captured imagery). This reference signal isclipped at the 13.5 and 86.5 percentile levels (lower and upper clippinglimits) and added to a binary payload signal at an amplitude (relativeto the reference signal) of 0.55.

This binary payload signal is formed by applying a payload (e.g., of 50bits), together with associated CRC data (e.g. of 25 bits), to aconvolutional encoder to produce a 1024 bit signature. This signature israndomized by XORing with a 1024 bit scrambling key. Each of theresulting bits is modulated by XORing with a fixed 16-bit spreading keysequence, transforming each of the randomized signature bits into 16binary “chips,” yielding 16,384 such chips. Each chip is assigned adifferent location in a 128×128 waxel array by data in a scatter table.At waxel locations where a “1” chip is mapped, the value of theunderlying signal (e.g., a flat grey signal of constant value 128) isincreased; at waxel locations where a “0” chip is mapped, the value ofthe underlying signal is decreased.

The combined signal that results from summation of these payload chipswith the reference signal values may have a resolution of 75 watermarkelements (waxels) per inch (WPI). A Gaussian bump shaping function isthen applied to up-sample the 75 WPI tile to the desired DPI. Forexample, to up-sample to a 300 DPI embedding resolution, the following4×4 bump shaping function can be applied:

$B = \begin{bmatrix}1 & 4 & 4 & 1 \\4 & {16} & {16} & 4 \\4 & {16} & {16} & 4 \\1 & 4 & 4 & 1\end{bmatrix}$If T₁₂₈ is the 75 WPI tile, the 300 DPI tile is obtained as followsT₅₁₂=T₁₂₈⊗B, where ⊗ is the Kronecker tensor product. Finally, theamplitude of the 128×128 waxel block is scaled to the desired rangedepending on the embedding strength. (In the present correlation-basedembedding case, the amplitude of the signal block is not relevant,except for rounding errors.)

FIG. 56 depicts examples of images carrying machine-readable codes.

The image on the far left is formed by placing or removing markingscomprised of a regular pattern according to a machine-readable, digitalwatermark signal. In this case, the regular pattern consists ofhorizontal rows/columns of spaced apart dark blocks, but a variety ofgraphic primitives may be used as this base pattern. These primitivesmay be squares, circles, triangles, or various other shapescorresponding to mark or no-mark areas when applied to a physical objectby printing, embossing, engraving, ablating, etching, thermal marking,or the like (e.g., ink or no ink; or mark or no mark on an object,including within one or more color separations applied to the object).To generate the data carrying image, the image formation processmodifies the pattern according to the digital watermark signal. This canbe implemented as a screening or masking process. Here, screening refersto forming the composite signal from screen elements, which are graphicprimitives. Masking refers to removing the base pattern according tobinarized watermark signal (locations where the watermark is a binary 1value retain the base pattern, whereas locations where the watermarkvalue is a binary 0 value are removed, or vice versa). For example, thedark square graphic primitive elements fill in regions where thewatermark signal is below a darkness level, or the absence of marks fillin regions whether the watermark is above a darkness level (asrepresented by brightness, grayscale, luminance value, or the like).This process generates a composite image comprising parts of the patternof graphic primitives and light areas that together form amachine-readable digital watermark signal. This composite image carriesthe digital watermark because the placing of the pattern, or converselythe shaping of the non-marked areas in the pattern, form the datecarrying signal.

The image in the center depicts an example of an image in which adigital watermark signal is conveyed in the line breaks of the line artpattern. In this case, the pattern formed of line and line breaks conveythe digital watermark signal. Again, dark elements are placedcorresponding to dark elements of the digital watermark signal in ascreening or masking method to place line elements or insert linebreaks.

The two images on the far right depict a third example image, shown atdifferent scales. This third image is formed from first and secondsource images, each encoded with distinct machine-readable signals. Thefirst and second images are combined to form a composite image in whichboth the machine-readable signals are readable. A first source imagecarries a first machine readable data payload encoded at a firstresolution (e.g., as quantified in cells per inch of data carrying cellswithin a tile). A second source image carries a second machine readablecode at a second, higher resolution. In this example, the first image isa binarized digital watermark image generated at a resolution of 75watermark cells per inch. The second source image is a Voronoi patternencoded with a digital watermark at 150 watermark cells per inch.Various of the binarized watermark signal and signal rich patternsdescribed in this document and incorporated documents may be used forthe first and second source images (e.g., Voronoi, stipple, Delaunay, orother stylized data carrying artwork). These two images are combinedusing an image editing operation in cells at the higher cells per inchresolution in which the markings in the cell of the first image arereplaced with or substituted for markings in corresponding celllocations (e.g., corresponding X, Y pixel coordinates) of the secondimage.

This approach is a method of creating a counterpart to a target image,the target image comprising two distinct 2D machine-readable codes. Thecounterpart is a composite image made from two source images. Thiscomposite image has an attribute of being decodable by a compliantreader apparatus to produce a plural-symbol payload earlier encodedtherein, and in particular, two different plural-symbol payloadsoriginating in the two source images.

The method generates a first image at a first resolution, the firstimage carrying a first machine-readable code. As noted, this may beaccomplished by binarizing a digital watermark signal, or digitallywatermarked image, using the various methods described in this orincorporated documents.

The method also generates a second image at a second resolution higherthan the first resolution, the second image carrying a second machinereadable code. Likewise, this may be accomplished with a binarizeddigital watermark or watermarked image, including the approaches ofgenerating stylized artwork bearing a digital watermark using Voronoi,stipple, Delaunay, style transfer, and related masking or screeningmethods described or incorporated in this document.

The method inserts parts of the second image into the first image tocreate a composite image. In one approach, it substitutes parts of thesecond image for parts of the first image at corresponding locations inthe first and second image where both the first and second images have amarking. This can be accomplished, for example, by an element-wiselogical AND operation of binarized versions of the two source images,where the elements are spatial cells of each image in a 2D pixel arrayat the resolution of the higher resolution image. It may also beaccomplished by screening a first source image with the primitiveelements in the cells of the higher resolution source image. It may alsobe accomplished by a masking operation (e.g., masking one source imagewith the other) such that the high resolution marked cells of the secondsource are placed within the lower resolution marked cells of the firstsource image. This method generates a composite image in which the firstand second machine-readable codes are decodable from an image capturedof an object bearing the composite image. Inverse operations may also beperformed with un-marked elements to make light regions within a darkbackground.

These methods for plural codes in one image are particularly suitablefor source images that are vector graphics, such as line art. Forexample, the second image parts comprise vector graphics that aresubstituted for artwork of the first image. Both the binarized firstsource image (e.g., a binarized digital watermark or digitallywatermarked image) and second source image may comprise vector graphicartwork.

As noted, the combining of the first and second source images maybeimplemented in variety of ways. One approach is the substitutingelements from the second source into the first with a pixel wise ANDoperation at the corresponding locations. These corresponding locationsare corresponding pixel coordinates in the first and second images.

The method applies for any embodiments where the first image comprises abinarized watermark signal and the second image comprises artworkcarrying a watermark signal at a resolution higher than the binarizedwatermark signal. For example, the second image may comprise a stylizedimage of data carrying graphical elements. These data carrying elementsmay be a stipple pattern, Voronoi pattern, Delaunay pattern, or othersignal rich art method, including style transfer method and its variantsdescribed herein.

The composite image may comprise two distinct machine-readable codesfrom the first and second images that are machine readable from imagescaptured at corresponding first and second depth ranges, the first depthrange comprising a maximum distance of a camera from a marked objectbearing the composite image that is greater than a maximum distance inthe second depth range. This is achieved by encoding the plural machinecodes in the source images at different spatial resolutions. Relativelylower resolution codes are readable at a depth range spanning at leastsome further distances than relatively higher resolution codes, whichare readable at depth range spanning at least closer distances betweenthe image sensor and marked object. Of course, the depth ranges may haveoverlapping distances.

For more examples and explanation of the examples in FIG. 56, please seeAppendix B, filed with application 62/972,522.

The processes and arrangements disclosed in this specification can beimplemented as instructions for computing devices, including generalpurpose processor instructions for a variety of programmable processors,such as microprocessors and systems on a chip (e.g., the Intel Atom, theARM A8 and Cortex series, the Qualcomm Snapdragon, and the nVidia Tegra4. Implementation can also employ a variety of specialized processors,such as graphics processing units (GPUs, such as are included in thenVidia Tegra series, and the Adreno 530—part of the Qualcomm Snapdragonprocessor), and digital signal processors (e.g., the Texas InstrumentsTMS320 and OMAP series devices, and the ultra-low power Qualcomm Hexagondevices, such as the QDSP6V5A), etc. These instructions can beimplemented as software, firmware, etc. These instructions can also beimplemented in various forms of processor circuitry, includingprogrammable logic devices, field programmable gate arrays (e.g., theXilinx Virtex series devices), field programmable object arrays, andapplication specific circuits—including digital, analog and mixedanalog/digital circuitry. Execution of the instructions can bedistributed among processors and/or made parallel across processorswithin a device or across a network of devices. Processing of data canalso be distributed among different processor and memory devices. Cloudcomputing resources can be used as well. References to “processors,”“modules” or “components” should be understood to refer tofunctionality, rather than requiring a particular form ofimplementation.

Implementation can additionally, or alternatively, employ specialpurpose electronic circuitry that has been custom-designed andmanufactured to perform some or all of the component acts, as anapplication specific integrated circuit (ASIC). Additional detailsconcerning special purpose electronic circuitry are provided in our U.S.Pat. No. 9,819,950.

Software instructions for implementing the detailed functionality can beauthored by artisans without undue experimentation from the descriptionsprovided herein, e.g., written in C, C++, Visual Basic, Java, Python,Tcl, Perl, Scheme, Ruby, Matlab, etc., in conjunction with associateddata.

Software and hardware configuration data/instructions are commonlystored as instructions in one or more data structures conveyed bytangible media, such as magnetic or optical discs, memory cards, ROM,etc., which may be accessed across a network. Some embodiments may beimplemented as embedded systems—special purpose computer systems inwhich operating system software and application software areindistinguishable to the user (e.g., as is commonly the case in basiccell phones). The functionality detailed in this specification can beimplemented in operating system software, application software and/or asembedded system software.

Different of the functionality can be implemented on different devices.Different tasks can be performed exclusively by one device or another,or execution can be distributed between devices. In like fashion,description of data being stored on a particular device is alsoexemplary; data can be stored anywhere: local device, remote device, inthe cloud, distributed, etc.

Details concerning watermarking included in embodiments of the presenttechnology are detailed in certain of the references earlier-identified,and are also disclosed in applicant's U.S. Pat. Nos. 6,122,403,6,590,996, 6,614,914, 6,975,744, 9,747,656, 9,959,587, 10,242,434,20170024840, 20190266749, 20190306385 and 20160364623.

This specification has discussed various embodiments. It should beunderstood that the methods, elements and concepts detailed inconnection with one embodiment can be combined with the methods,elements and concepts detailed in connection with other embodiments.While some such arrangements have been particularly described, many havenot—due to the number of permutations and combinations. Applicantsimilarly recognizes and intends that the methods, elements and conceptsof this specification can be combined, substituted and interchanged—notjust among and between themselves, but also with those known from thecited prior art. Moreover, it will be recognized that the detailedtechnology can be included with other technologies—current andupcoming—to advantageous effect. Implementation of such combinations isstraightforward to the artisan from the teachings provided in thisdisclosure.

While this disclosure has detailed particular orderings of acts andparticular combinations of elements, it will be recognized that othercontemplated methods may re-order acts (possibly omitting some andadding others), and other contemplated combinations may omit someelements and add others, etc.

Although disclosed as complete systems, sub-combinations of the detailedarrangements are also separately contemplated (e.g., omitting various ofthe features of a complete system).

While certain aspects of the technology have been described by referenceto illustrative methods, it will be recognized that apparatusesconfigured to perform the acts of such methods are also contemplated aspart of applicant's inventive work. Likewise, other aspects have beendescribed by reference to illustrative apparatus, and the methodologyperformed by such apparatus is likewise within the scope of the presenttechnology. Still further, tangible computer readable media containinginstructions for configuring a processor or other programmable system toperform such methods is also expressly contemplated.

To provide a comprehensive disclosure, while complying with the PatentAct's requirement of conciseness, applicant incorporates-by-referenceeach of the documents referenced herein. (Such materials areincorporated in their entireties, even if cited above in connection withspecific of their teachings.) These references disclose technologies andteachings that applicant intends be incorporated into the arrangementsdetailed herein, and into which the technologies and teachingspresently-detailed be incorporated.

In view of the wide variety of embodiments to which the principles andfeatures discussed above can be applied, it should be apparent that thedetailed embodiments are illustrative only, and should not be taken aslimiting the scope of the invention.

The invention claimed is:
 1. A method of producing an output 2Dmachine-readable indicia from an input 2D machine-readable indicia,comprising the acts: identifying a first excerpt of the input 2Dmachine-readable indicia; processing the first excerpt to yield a tilehaving an organized structure, by applying to said first excerpt one ormore transformation operations, including at least one transformationoperation from the list: (a) mirroring, (b) rotating, (c) translation,(d) toroidal shifting, (e) signal-inverting, (f) scaling and (g) glidetransformation; incorporating said tile in the output 2Dmachine-readable indicia; and rendering said output 2D machine-readableindicia on a tangible article to be visible to human observers, whereinthe output indicia has a non-steganographic form on said article.
 2. Themethod of claim 1 that further includes: capturing imagery of saidtangible article; decoding a plural-symbol payload from a depiction ofthe output indicia in the captured imagery; and taking an action basedon the decoded plural-symbol payload.
 3. The method of claim 2 in whichthe action includes adding said article to a checkout tally of ashopper.
 4. The method of claim 1 that further includes applying to saidfirst excerpt two different transformation operations from said list. 5.The method of claim 1 that further includes pre-processing the input 2Dmachine-readable indicia prior to identifying the first excerpttherefrom, said pre-processing including applying a conformaltransformation to at least part of the input 2D machine-readableindicia, and applying an inverse-conformal transformation to the tileprior to incorporating the tile into the output indicia.
 6. The methodof claim 1 in which the output 2D machine-readable indicia is other thanan affine-transformed version of the input 2D machine-readable indicia.7. The method of claim 1 in which the output 2D machine-readable indiciais other than a toroidally-shifted version of the input 2Dmachine-readable indicia.
 8. The method of claim 1 in which the output2D machine-readable indicia is other than plural versions of the input2D machine-readable indicia summed at different scales.
 9. The method ofclaim 1 in which said processing comprising applying one of said listedtransformation operations to the first excerpt to yield a first interimresult, applying one of said listed transformation operations to thefirst excerpt to yield a second interim result different than the firstresult, and summing the first and second interim results.
 10. The methodof claim 1 that further includes: identifying a second excerpt of theinput 2D machine-readable indicia, different than the first excerpt;processing the second excerpt to yield a second tile having an organizedstructure, by applying to said second excerpt one or more transformationoperations from said list; and incorporating said second tile in theoutput 2D machine-readable indicia.
 11. The method of claim 1 in whichsaid rendering of the output indicia in non-steganographic form allowsrelaxed rendering tolerances as compared with an output indicia that isto be rendered in steganographic form.
 12. The method of claim 1 inwhich said tile has an axis of symmetry.
 13. The method of claim 1 inwhich said one or more transformation operations comprises rotating theexcerpt by plural different angles, or shifting the excerpt by pluraldifferent shifts, to thereby yield plural intermediate results, and saidprocessing includes computing a sum of said intermediate results. 14.The method of claim 13 in which said processing includes assigning aweighting factor to each of said plural intermediate results, scalingeach of said intermediate results by the weighting factor assignedthereto, and summing the scaled intermediate results to yield a weightedsum.
 15. The method of claim 14 wherein at least one of said weightingfactors is a negative number.
 16. The method of claim 1 in whichidentifying the first excerpt of the input 2D machine-readable indiciaincludes applying a windowing function to the input 2D machine-readableindicia, or a sub-block thereof.
 17. The method of claim 1 in which theexcerpt has a non-square shape.
 18. A method of producing an output 2Dmachine-readable indicia from an input 2D machine-readable indicia,comprising the acts: identifying an excerpt of the input 2Dmachine-readable indicia; processing the excerpt to yield a tile havingan organized structure, by applying to said excerpt two or moretransformation operations, including at least two differenttransformation operations from the list: (a) mirroring, (b) rotating,(c) translation, (d) toroidal shifting, (e) signal-inverting, (f)scaling, and (g) glide transformation; incorporating said tile in theoutput 2D machine-readable indicia.
 19. The method of claim 18 thatfurther includes: forming said output 2D machine-readable indicia on atangible article; capturing imagery of said tangible article; decoding aplural-symbol payload from a depiction of the output indicia in thecaptured imagery; and taking an action based on the decodedplural-symbol payload.
 20. A method of producing an output 2Dmachine-readable indicia from an input 2D machine-readable indicia,comprising the acts: identifying an excerpt of the input 2Dmachine-readable indicia; processing the excerpt to yield a tile havingan organized structure, by: (i) applying to said excerpt atransformation operation from the list: (a) mirroring, (b) rotating, (c)translation, (d) toroidal shifting, (e) signal-inverting, and (f) glidetransformation, to yield a first interim result; (ii) applying to saidexcerpt a transformation operation from said list to yield a secondinterim result different than the first interim result; and (iii)summing the first and second interim results; and incorporating saidtile in the output 2D machine-readable indicia.